ImpactMojo ImpactMojo
Premium

Data Literacy 101

Data Sources, Quality Assessment & Open Data
ImpactMojo Workshop Series • Finding & Evaluating Reliable Information
75-90 Minutes

Workshop 2: Source Evaluation & Data Quality

Target Audience: Development practitioners who need to find and evaluate data sources for reports, proposals, and evidence-based programming

Prerequisites: Workshop 1 or basic data reading skills

Materials Needed: Internet access, sample datasets, government reports

Learning Objectives

By the end of this workshop, participants will be able to:

Part 1: The Data Source Hierarchy - Not All Data is Created Equal

20 minutes

The Same "Fact" from Different Sources

Question: What percentage of Indian women participate in the labor force?

Source A: WhatsApp forward - "Only 12% of Indian women work - shocking!"

Source B: News article - "Women's labor force participation drops to 16.5% in 2022"

Source C: PLFS 2022-23 - "Labour Force Participation Rate for women: 37%"

Source D: ILO Database - "Female labor force participation (ages 15+): 24.8%"

Why the differences?

  • Different definitions (work vs. labor force participation vs. employment)
  • Different age groups (15+ vs. working age)
  • Different time periods and survey methods
  • Urban vs. rural vs. total population

This is why source evaluation matters - the "same" statistic can vary dramatically based on methodology.

Data Source Reliability Hierarchy

Unreliable Questionable Generally Reliable Highly Reliable
Unreliable Sources
  • Social media posts without sources
  • Unverified WhatsApp forwards
  • Blogs without citations
  • Outdated reports (>10 years old)
  • Clearly biased advocacy materials

Use: Never for serious analysis

Questionable Sources
  • News articles without data citations
  • Think tank reports with unclear methods
  • Corporate/marketing research
  • Small-scale studies without peer review
  • International rankings without transparency

Use: With extreme caution, verify elsewhere

Generally Reliable
  • Government statistical reports
  • UN agency publications
  • Academic research with methods
  • Established NGO research reports
  • Well-documented survey data

Use: Good for most purposes, check currency

⭐ Highly Reliable
  • Peer-reviewed academic papers
  • Government census/official surveys
  • International statistical databases
  • Replicated research findings
  • Meta-analyses and systematic reviews

Use: Gold standard for evidence-based work

Part 2: Quality Assessment Framework

25 minutes

The 8-Point Data Quality Checklist

Quality Dimension Key Questions Red Flags
1. Currency How recent is this data? Is it still relevant? Data >5 years old for rapidly changing topics
2. Coverage Who/what is included? What's excluded? Unclear sample, convenience sampling
3. Methodology How was data collected? What methods used? No methodology section, vague descriptions
4. Sample Size Is the sample large enough to be representative? Very small samples, no confidence intervals
5. Transparency Are limitations acknowledged? Data available? No access to raw data, missing details
6. Independence Who funded this? Any conflicts of interest? Industry-funded studies, undisclosed conflicts
7. Consistency Do findings align with other credible sources? Wildly different from established evidence
8. Precision Are uncertainty ranges provided? Margins of error? Overly precise claims, no confidence intervals

Source Evaluation Exercise (15 minutes)

Your Task: Evaluate three data sources using the 8-point checklist

Source A: "India Healthcare Report 2024"
  • Publisher: HealthConsult Pvt Ltd (consulting firm)
  • Sample: "500+ healthcare professionals across metros"
  • Key Finding: "85% of doctors report improved patient outcomes with new technology"
  • Methodology: "Online survey conducted in Q1 2024"
  • Funding: Not disclosed

Rate Source A (1-5 scale) on each dimension:

  • Currency: ___/5
  • Coverage: ___/5
  • Methodology: ___/5
  • Sample Size: ___/5
  • Transparency: ___/5
  • Independence: ___/5
  • Consistency: ___/5
  • Precision: ___/5

Overall Assessment: Would you use this source? Why or why not?

Source B: "National Sample Survey 78th Round: Health in India"
  • Publisher: National Statistical Office, Ministry of Statistics
  • Sample: "1,13,823 households across all states and UTs"
  • Time Period: July 2020 - June 2021
  • Methodology: "Two-stage stratified sampling, detailed in annexure"
  • Limitations: "COVID-19 impact on field work noted, adjustments made"

Quick Assessment: How would Source B likely score compared to Source A? Discuss in pairs.

Part 3: Navigating Indian Data Ecosystem

20 minutes

Key Government Data Sources

Central Statistics Office

What: NSS, NSSO surveys, national accounts

Best For: Household consumption, employment, health

Frequency: Annual, quinquennial

Access: mospi.gov.in

Census of India

What: Population, housing, economic census

Best For: Demographics, literacy, housing

Frequency: Every 10 years

Access: censusindia.gov.in

ASER & AISHE

What: Learning outcomes, higher education

Best For: Education quality, enrollment trends

Frequency: Annual

Access: asercentre.org, aishe.gov.in

NFHS & HMIS

What: Health, nutrition, family welfare

Best For: Health outcomes, service delivery

Frequency: Every 3-5 years / Continuous

Access: dhsprogram.com, hmis.nhp.gov.in

Agricultural Statistics

What: Crop production, prices, farmer income

Best For: Rural livelihoods, food security

Frequency: Annual, seasonal

Access: eands.dacnet.nic.in

RBI & Financial Data

What: Financial inclusion, credit, SHG data

Best For: Economic indicators, banking

Frequency: Monthly, quarterly

Access: rbi.org.in, pmjdy.gov.in

Understanding Administrative Data Limitations:

  • Coverage gaps: May miss informal sector, marginalized groups
  • Reporting bias: Incentives for officials to over/under-report
  • Definitional changes: Categories may change over time
  • Time lags: Often 1-2 years behind current reality
  • Quality variation: Data quality varies by state/district

Part 4: Problem Set - Source Selection & Quality Assessment

20 minutes

Research Scenario: Data Source Selection

Background: You're writing a report on "Digital Financial Inclusion in Rural India" for an international development agency. You need credible, recent data to support policy recommendations.

Problem 1: Source Identification (8 minutes)

Research Questions to Answer:

  1. What percentage of rural Indians have bank accounts?
  2. How has digital payment adoption changed post-COVID?
  3. What are the main barriers to financial inclusion for women?
  4. Which states have made most progress on financial inclusion?

For each question, identify:

  1. Best primary data source: _____
  2. Alternative/secondary source: _____
  3. Potential data limitations to mention: _____
  4. How recent should the data be: _____

Available Sources: NFHS-5, Financial Inclusion Insights Survey, RBI Annual Report, NABARD All India Rural Financial Inclusion Survey, State-wise PMJDY data, Digital India reports

Problem 2: Quality Assessment Challenge (7 minutes)

Source under Review: "India Digital Payment Growth Report 2024" by TechFinance Solutions

Key Claims:

  • "Digital payments grew 400% in rural areas since 2020"
  • "76% of farmers now use UPI for transactions"
  • "Women's digital payment adoption lags by only 5%"

Available Information:

  • Sample: 2,500 respondents across 12 states
  • Method: Phone interviews (mobile numbers from telecom databases)
  • Funding: Supported by major fintech companies
  • Response rate: Not mentioned
  • Sampling method: Not clearly described

Critical Assessment:

  1. Identify 3 quality red flags:
    • _____
    • _____
    • _____
  2. What sampling bias might exist with phone interviews? _____
  3. How might funding by fintech companies affect findings? _____
  4. Would you cite this report? How would you caveat it? _____
Problem 3: Data Triangulation (5 minutes)

Scenario: You find conflicting statistics on women's bank account ownership:

  • Global Findex 2021: 78% of women have accounts
  • NFHS-5: 53% of women have accounts they use themselves
  • PMJDY Dashboard: 56% of PMJDY accounts belong to women

Resolution Strategy:

  1. Why might these numbers differ? (Consider definitions, time periods, samples)
  2. Which would you use for what purposes?
  3. How would you present this in your report?

Reflection & Resource Building

10 minutes

Personal Data Source Library

Individual Task (7 minutes):

Create your go-to source list for your work area:

  1. Your main focus area: _____ (e.g., education, health, livelihoods)
  2. Top 3 most reliable sources for this area:
    • _____
    • _____
    • _____
  3. One source you used to trust but now question: _____
  4. Biggest data gap in your field: _____
  5. One new source you want to explore: _____

Quick Share (3 minutes): Exchange source recommendations with colleagues in similar fields

Key Takeaway

Good data literacy means being a skeptical consumer of information. Always ask: Who collected this? How? Why? What might be missing? And never rely on a single source for important decisions.

Source Evaluation Resources

Fact-Checking Organizations:

Data Quality Guides:

Indian Government Data Portals:

Next Steps in ImpactMojo:

This handout is part of the ImpactMojo 101 Knowledge Series
Licensed under CC BY-NC-ND 4.0 • Free to use with attribution • www.impactmojo.in

For source evaluation checklists, data portal guides, and quality assessment templates, visit the ImpactMojo platform.